Inter-reviewer conflict resolution

ABSTRACT

One embodiment provides a method, including: receiving a plurality of review comments from each of a plurality of reviewers tasked with reviewing a document; categorizing each of the plurality of review comments into one of a plurality of review topics; identifying a conflict between a first review comment provided by one of the plurality of reviewers and a second review comment provided by another of the plurality of reviewers, wherein the identifying a conflict comprises (i) identifying a sentiment of the first review comment and a sentiment of the second review comment and (ii) determining that the sentiment of the first review comment and the sentiment of the second review comment are different; and generating a question set comprising a plurality of questions based upon a conflict identified for a review comment of the corresponding reviewer, wherein the corresponding reviewer answering the generated question resolves the conflict.

BACKGROUND

When documents (e.g., technical papers, grant proposals, manuscripts,presentations, etc.) are prepared by one or a group of people, they aregenerally reviewed by one or more reviewers. Sometimes each of thereviewers is responsible for reviewing a certain feature or aspect ofthe document. For example, one or a group of reviewers may beresponsible for reviewing the document for technical accuracy, whereasanother reviewer or group of reviewers is responsible for reviewing thedocument for novelty. Alternatively, all of the reviewers may beresponsible for reviewing all aspects or features of the document.

Each reviewer may have a different style of reviewing and/or may expectdifferent features or aspects to be presented in a particular manner.For example, one reviewer may provide detailed comments or questions onaspects of the document, whereas another reviewer may provide shortcomments without much substance. As another example, one reviewer mayexpect all related documents or works to be cited within a document,whereas another reviewer may only expect the most closely relateddocuments or works to be cited within a document. As another example,one reviewer may provide more subjective comments, whereas anotherreviewer may provide more objective comments.

BRIEF SUMMARY

In summary, one aspect of the invention provides a method comprising:receiving a plurality of review comments from each of a plurality ofreviewers tasked with reviewing a document; categorizing, for each ofthe plurality of reviewers, each of the plurality of review commentsinto one of a plurality of review topics; identifying a conflict between(a) a first review comment categorized into a review topic and providedby one of the plurality of reviewers and (b) a second review commentcategorized into the same review topic and provided by another of theplurality of reviewers, wherein the identifying a conflict comprises (i)identifying a sentiment of the first review comment and a sentiment ofthe second review comment and (ii) determining that the sentiment of thefirst review comment and the sentiment of the second review comment aredifferent; and generating, for each of the plurality of reviewers, aquestion set comprising a plurality of questions based upon a conflictidentified for a review comment of the corresponding reviewer, whereinthe corresponding reviewer answering the generated question resolves theconflict.

Another aspect of the invention provides an apparatus, comprising: atleast one processor; and a computer readable storage medium havingcomputer readable program code embodied therewith and executable by theat least one processor, the computer readable program code comprising:computer readable program code configured to receive a plurality ofreview comments from each of a plurality of reviewers tasked withreviewing a document; computer readable program code configured tocategorize, for each of the plurality of reviewers, each of theplurality of review comments into one of a plurality of review topics;computer readable program code configured to identify a conflict between(a) a first review comment categorized into a review topic and providedby one of the plurality of reviewers and (b) a second review commentcategorized into the same review topic and provided by another of theplurality of reviewers, wherein the identifying a conflict comprises (i)identifying a sentiment of the first review comment and a sentiment ofthe second review comment and (ii) determining that the sentiment of thefirst review comment and the sentiment of the second review comment aredifferent; and computer readable program code configured to generate,for each of the plurality of reviewers, a question set comprising aplurality of questions based upon a conflict identified for a reviewcomment of the corresponding reviewer, wherein the correspondingreviewer answering the generated question resolves the conflict.

An additional aspect of the invention provides a computer programproduct, comprising: a computer readable storage medium having computerreadable program code embodied therewith, the computer readable programcode executable by a processor and comprising: computer readable programcode configured to receive a plurality of review comments from each of aplurality of reviewers tasked with reviewing a document; computerreadable program code configured to categorize, for each of theplurality of reviewers, each of the plurality of review comments intoone of a plurality of review topics; computer readable program codeconfigured to identify a conflict between (a) a first review commentcategorized into a review topic and provided by one of the plurality ofreviewers and (b) a second review comment categorized into the samereview topic and provided by another of the plurality of reviewers,wherein the identifying a conflict comprises (i) identifying a sentimentof the first review comment and a sentiment of the second review commentand (ii) determining that the sentiment of the first review comment andthe sentiment of the second review comment are different; and computerreadable program code configured to generate, for each of the pluralityof reviewers, a question set comprising a plurality of questions basedupon a conflict identified for a review comment of the correspondingreviewer, wherein the corresponding reviewer answering the generatedquestion resolves the conflict.

A further aspect of the invention provides a method, comprising:obtaining a plurality of document evaluation comments from each of aplurality of evaluators corresponding to evaluation of a document;classifying each of the evaluation comments into one of a plurality ofevaluation topics, wherein the classifying comprises extracting phrasesfrom the evaluation comments and classifying the extracted phrases;detecting, for each of the evaluation topics, a disagreement between afirst evaluation comment provided by one of the plurality of reviewersand a second evaluation comment provided by another of the plurality ofreviewers, wherein the detecting comprises detecting a contrastingsentiment between the first and second evaluation comments; generating,for each of the plurality of evaluators, a minimal question set toresolve the disagreement between the evaluation comments; and providingthe minimal question set to the corresponding evaluator.

For a better understanding of exemplary embodiments of the invention,together with other and further features and advantages thereof,reference is made to the following description, taken in conjunctionwith the accompanying drawings, and the scope of the claimed embodimentsof the invention will be pointed out in the appended claims.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates a method of resolving inter-reviewer conflicts bygenerating a minimal question set for each of the reviewers.

FIG. 2 illustrates a computer system.

DETAILED DESCRIPTION

It will be readily understood that the components of the embodiments ofthe invention, as generally described and illustrated in the figuresherein, may be arranged and designed in a wide variety of differentconfigurations in addition to the described exemplary embodiments. Thus,the following more detailed description of the embodiments of theinvention, as represented in the figures, is not intended to limit thescope of the embodiments of the invention, as claimed, but is merelyrepresentative of exemplary embodiments of the invention.

Reference throughout this specification to “one embodiment” or “anembodiment” (or the like) means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the invention. Thus, appearances of thephrases “in one embodiment” or “in an embodiment” or the like in variousplaces throughout this specification are not necessarily all referringto the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in at least one embodiment. In thefollowing description, numerous specific details are provided to give athorough understanding of embodiments of the invention. One skilled inthe relevant art may well recognize, however, that embodiments of theinvention can be practiced without at least one of the specific detailsthereof, or can be practiced with other methods, components, materials,et cetera. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringaspects of the invention.

The illustrated embodiments of the invention will be best understood byreference to the figures. The following description is intended only byway of example and simply illustrates certain selected exemplaryembodiments of the invention as claimed herein. It should be noted thatthe flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, apparatuses, methods and computer program products accordingto various embodiments of the invention. In this regard, each block inthe flowchart or block diagrams may represent a module, segment, orportion of code, which comprises at least one executable instruction forimplementing the specified logical function(s).

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or combinations of special purpose hardware and computerinstructions.

Specific reference will be made here below to FIGS. 1-2. It should beappreciated that the processes, arrangements and products broadlyillustrated therein can be carried out on, or in accordance with,essentially any suitable computer system or set of computer systems,which may, by way of an illustrative and non-restrictive example,include a system or server such as that indicated at 12′ in FIG. 2. Inaccordance with an example embodiment, all of the process steps,components and outputs discussed with respect to FIG. 1 can be performedor utilized by way of a processing unit or units and system memory suchas those indicated, respectively, at 16′ and 28′ in FIG. 2, whether on aserver computer, a client computer, a node computer in a distributednetwork, or any combination thereof.

Since different reviewers provide comments to documents differently, itcan be difficult for the person or group of people receiving thecomments to determine what changes need to be made in order to resolvethe comments. For example, a reviewer may indicate that the documentdoes not include novel features, but does not indicate what novelfeatures are missing or why the reviewer thinks that the novel featuresare missing. In other words, the reviewer may provide an overall generalcomment without any supporting details explaining the reasoning behindthe comment. Additionally, a reviewer could provide comments thatconflict with comments provided by other reviewers. For example, onereviewer may indicate that the document accurately captures thetechnical details, and another reviewer may indicate that the documentfails to capture all the technical details.

To resolve these inter-reviewer conflicts conventionally requiresback-and-forth between the reviewers and/or the document drafter. Thisprocess can be very time consuming and requires a significant effortfrom all parties. For example, it may be difficult to get all partiestogether to communicate in a single session, so the communications mustoccur through another medium that requires multiple hours or days toresolve. Additionally, if the reviewers have to communicate through atextual medium, it may take many communications to fully convey orunderstand a point being made by the reviewer. Thus, the conventionaltechnique for resolving comments is very time consuming, laborintensive, and inefficient.

Accordingly, the techniques and systems as described herein provide asystem and technique for resolving inter-reviewer conflicts bygenerating a minimal question set for each of the reviewers that, uponreceiving responses to the question set, will resolve the conflicts. Thesystem receives a plurality of comments from each of a plurality ofreviewers that are tasked with reviewing a document. The system can thencategorize the comments into one of a plurality of review topics, forexample, novelty, technical details, relevance, or the like. The systemmay further categorize the comments into finer-grained topics, forexample, novelty 1, novelty 2, or the like.

The system can then identify conflicts between comments provided bydifferent reviewers that are categorized into the same topic. Forexample, the system may analyze the sentiment of each of the comments todetermine whether the comment has a positive, negative, or neutralsentiment. If the sentiments of two or more comments are differentacross reviewers, the system may determine that there is a conflictbetween the review comments provided by the reviewers. Identification ofa conflict may also include identifying comments provided by reviewersthat are lacking in substance. In other words, a conflict may beidentified if a comment is a subjective comment without any supportingreasoning for the provided comment. From the identified conflicts andsubjective comments, the system can automatically generate a minimalquestion set for each of the reviewers. These question sets aregenerated in a manner such that an answer to the question set resolvesthe conflicts between the reviewers. In other words, the systemautomatically generates the least number of questions for each of thereviewers that results in resolution of the conflicts once the reviewerhas provided a response to each of the questions in the question set.

Such a system provides a technical improvement over current systems fordocument review by providing a technique that can automatically generatea minimal question set for each reviewer, where reviewer responses tothe question set will resolve any conflicts between reviewer comments.The system categorizes comments made by reviewers and can identifyconflicts between comments made by different reviewers. Based upon theseconflicts, which also include comments that need additional detail, thesystem can automatically generate natural language questions to bepresented to each reviewer that are designed to resolve the conflicts.Thus, instead of the conventional system which is time consuming, laborextensive, and inefficient, the described system provides an efficienttechnique for resolution of reviewer comments.

FIG. 1 illustrates a method for resolving inter-reviewer conflicts bygenerating a minimal question set for each of the reviewers that, uponreceiving responses to the question set, will resolve the conflicts. At101 the system receives a plurality of review comments from each of aplurality of reviewers who are tasked with reviewing a document. Inother words, the system receives multiple review comments from each ofthe document reviewers. Receiving the review comments may include eachof the reviewers or another user providing comments to a central system,uploading comments, or the like. Alternatively, receipt of the reviewcomments may include a user providing a document that includes reviewcomments to the system, for example, by uploading a reviewed/commenteddocument, providing access to the reviewed/commented document, or thelike.

At 102 the system may categorize each of the review comments for each ofthe reviewers. In other words, the system maintains identification ofthe reviewer who provided the comment and categorizes each of thecomments provided by a single reviewer. The system performs the samecategorization for each of the remaining reviewers. To categorize thecomments the system may first categorize the comments into one of aplurality of review topics. The review topics may be dependent on thedocument type. For example, a document for a grant proposal may includea funding topic, whereas a document for a technical presentation may notinclude a funding topic. Alternatively, the topics may be provided by auser and the system may simply categorize each of the comments basedupon the provided topics. Some possible high-level topics includenovelty, related words, results, relevance, writing style, mathematicalformulation, and the like. A user can also customize the topics basedupon the setting of the document, for example, the user may choosedifferent topics for documents being presented at a symposium versus adocument being submitted to a journal. If a single comment includesdifferent portions that appear to belong to different topics, the systemmay break up the comment into the different portions and categorize eachof those portions. Alternatively, the system may simply duplicate thecomment across the multiple topics so that the comment appears in allapplicable topics.

The system may also perform a fine-tuned categorization where each ofthe comments provided within a single high-level categorization arefurther classified or categorized into more specific or detailed topics.For example, the system may further classify or categorize commentsprovided in the novelty topic into technical or non-technical noveltytopics. In either the high-level categorization or the fine-tunedcategorization the system may use a deep learning technique to classifyphrases of the review comments into different topics. Thus, to performthe categorization the system may use natural language processing toidentify words and phrases within the comments and also to identifysyntactic and/or syntax roles of the words or phrases within thecomments. The system can then use either deep learning techniques ordictionary based matching to classify the identified words or phrasesinto different topics.

At 103 the system determines whether there is a conflict between reviewcomments of different reviewers. The conflict analysis is performedwithin the different topics. In other words, the system analyzes all thecomments provided by different reviewers that have been categorized intoa particular review topic. Within each of these topics, the systemdetermines if there are any conflicts between the comments provided bydifferent reviewers. To identify if any conflicts exist the systemperforms a sentiment analysis on each of the comments within the topicand categorizes the sentiment of the comment. For example, the systemmay categorize a comment as having a positive sentiment, a negativesentiment, a neutral sentiment, or the like. If the comment has multipleparts, each having a different sentiment, the system may break thecomment into different portions and then sentimentally categorize eachof the portions. Sentiment analysis may be performed using naturallanguage processing techniques and/or a sentiment model that is uniqueto the topic. In other words, the sentiment model that determines thesentiment of the comment may be unique to each topic so that a moreaccurate sentiment conclusion can be determined.

Once the system has performed the sentiment analysis for each of thecomments within the topic, the system compares the sentiment of commentswithin the topic provided by different reviewers. In other words, thesystem is trying to determine if different reviewers providedconflicting comments. Thus, the system compares the sentiment ofcomments within a topic provided by one reviewer against the sentimentof comments within the same topic provided by another reviewer. If thesentiment of the comments provided by reviewers is different, the systemmay determine that a conflict exists between the review commentsprovided by the different reviewers. For example, the system mayidentify a conflict if a review comment provided by one reviewer andcategorized into one topic has a negative sentiment and a review commentprovided by a different reviewer and categorized into the same topic hasa neutral sentiment. As another example, the system may identify aconflict if a review comment provided by one reviewer and categorizedinto one topic has a positive sentiment and a review comment provided bya different reviewer and categorized into the same topic has a negativesentiment.

The system may also identify a conflict within a single comment providedby a reviewer. An example of such a conflict is when the comment is asubjective comment that includes no supporting detail for the comment.In other words, if a reviewer provides an overall general comment butdoes not provide any reasoning or details for the comment, the systemcan identify this comment as having a conflict even though it is not indirect conflict with a comment provided by another reviewer. In order toidentify a comment that is subjective with little detail or reasoning,the system can generate a question from the comment. Generation of thequestion may be performed using deep learning methods or models. Oncethe question is generated, the system may analyze the comment todetermine if the answer to the question appears in the comment. In otherwords, the system can treat the comment as a question-answer generationproblem. If the answer does appear within the comment, the system candiscard that question. If answers appear to all generated questions,then the system can categorize the comment as a proper subjectivecomment having details and/or reasoning. If the comment does not includeanswers to some of the questions, the system can mark the comment asnon-explained and, therefore, as a comment having a conflict.

Another example of detection of a conflict which does not include adirect conflict with another reviewer's comment is a comment thatrequires extra validation. A comment that requires extra validation is acomment that indicates that something is missing, incorrect, or thelike, and requires a person to verify or validate whether theinformation is actually missing or incorrect. These extra validationconflicts can be identified using a deep learning method or model ornatural language processing technique to analyze the comment andidentify that the comment can be categorized as a validation comment.

If the system determines that there is not a conflict at 103, the systemmay discard the review comment for question generation at 105. In otherwords, if the system determines that a review comment does not needadditional input by a reviewer, the system will not use that comment forquestion generation.

If, on the other hand, the system determines there is a conflict at 103,the system will generate a question set based upon the conflict at 104.The system will generate a question set for each of the reviewers basedupon conflicts identified that correspond to the reviewer. In otherwords, the system generates unique question sets for each of thereviewers. The unique question sets may contain some of the samequestions across reviewers. However, a question set for one reviewerwill not be identical to a question set for another reviewer. The systemattempts to generate a minimal question set for each of the reviewers. Aminimal question set is a question set that has the fewest number ofquestions that are necessary for resolving the identified conflicts.This helps to reduce the number of questions presented to the reviewers,thereby providing a more efficient system. The question sets includesquestions where a response provided by the reviewer would resolve theidentified conflict. In other words, the generated questions are thosequestions necessary for resolving the conflict.

Generation of the minimal question set is based upon the conflictsidentified from the sentiment analysis, the subjective review analysis,and the extra validation analysis. In other words, the system generatesquestions that can resolve any of the conflicts identified from thesentiment analysis, subjective review analysis, and the extra validationanalysis. To generate the questions the system uses a deep learningmodel that is trained for question generation based upon the identifiedconflicts. From the model the system can generate questions using anatural language processing technique so that the questions are providedin a manner understandable by the reviewer, specifically, a naturallanguage question. Before presentation to the reviewers, the system mayreduce the questions to a subset of minimal questions.

To produce the minimal questions the system may first validate thegenerated questions against identified parameters. These parameters maybe based upon a submission setting for the document, for example, asymposium, a grant proposal, a presentation, a technical manual, or thelike. Each of the submission settings may have different features oraspects that are expected within a submitted document. For example, onesubmission setting may require more technical details than anothersubmission setting. Thus, based upon the submission setting, the systemmay determine that some of the questions are irrelevant. In other words,if a generated question would request additional details that are notnecessary for the submission setting, the system may identify thatquestion as irrelevant and discard that question. Another technique forreducing the number of questions may be to compare the questions againstother generated questions to determine if the answer to one of thequestions would also answer the other question. In this case, the systemmay identify one of the questions as redundant and discard thatquestion. The system may also compare questions generated acrossreviewers to determine if some of those questions would be redundant andmay then discard one of the redundant questions. In other words, if ananswer provided by one reviewer would answer a question generated foranother reviewer, the system may identify one of those questions asredundant and discard that question.

Thus, the described system and method provide an improvement overconventional system for document review by providing a system that canautomatically generate minimal question sets for each of the reviewerssuch that responses to the question sets resolve any conflicts orsubjective comments that were identified. Such a system provides atechnique that is less time consuming, less labor intensive, and moreefficient than conventional manual techniques that require manycommunications between reviewers and/or document drafters.

As shown in FIG. 2, computer system/server 12′ in computing node 10′ isshown in the form of a general-purpose computing device. The componentsof computer system/server 12′ may include, but are not limited to, atleast one processor or processing unit 16′, a system memory 28′, and abus 18′ that couples various system components including system memory28′ to processor 16′. Bus 18′ represents at least one of any of severaltypes of bus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) bus.

Computer system/server 12′ typically includes a variety of computersystem readable media. Such media may be any available media that areaccessible by computer system/server 12′, and include both volatile andnon-volatile media, removable and non-removable media.

System memory 28′ can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30′ and/or cachememory 32′. Computer system/server 12′ may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34′ can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18′ by at least one datamedia interface. As will be further depicted and described below, memory28′ may include at least one program product having a set (e.g., atleast one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40′, having a set (at least one) of program modules 42′,may be stored in memory 28′ (by way of example, and not limitation), aswell as an operating system, at least one application program, otherprogram modules, and program data. Each of the operating systems, atleast one application program, other program modules, and program dataor some combination thereof, may include an implementation of anetworking environment. Program modules 42′ generally carry out thefunctions and/or methodologies of embodiments of the invention asdescribed herein.

Computer system/server 12′ may also communicate with at least oneexternal device 14′ such as a keyboard, a pointing device, a display24′, etc.; at least one device that enables a user to interact withcomputer system/server 12′; and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 12′ to communicate withat least one other computing device. Such communication can occur viaI/O interfaces 22′. Still yet, computer system/server 12′ cancommunicate with at least one network such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20′. As depicted, network adapter 20′communicates with the other components of computer system/server 12′ viabus 18′. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12′. Examples include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

This disclosure has been presented for purposes of illustration anddescription but is not intended to be exhaustive or limiting. Manymodifications and variations will be apparent to those of ordinary skillin the art. The embodiments were chosen and described in order toexplain principles and practical application, and to enable others ofordinary skill in the art to understand the disclosure.

Although illustrative embodiments of the invention have been describedherein with reference to the accompanying drawings, it is to beunderstood that the embodiments of the invention are not limited tothose precise embodiments, and that various other changes andmodifications may be affected therein by one skilled in the art withoutdeparting from the scope or spirit of the disclosure.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions. These computer readable programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks. These computer readable program instructions may also be storedin a computer readable storage medium that can direct a computer, aprogrammable data processing apparatus, and/or other devices to functionin a particular manner, such that the computer readable storage mediumhaving instructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A method, comprising: receiving a plurality ofreview comments from each of a plurality of reviewers tasked withreviewing a document; categorizing, for each of the plurality ofreviewers, each of the plurality of review comments into one of aplurality of review topics; identifying a conflict between (a) a firstreview comment categorized into a review topic and provided by one ofthe plurality of reviewers and (b) a second review comment categorizedinto the same review topic and provided by another of the plurality ofreviewers, wherein the identifying a conflict comprises (i) identifyinga sentiment of the first review comment and a sentiment of the secondreview comment and (ii) determining that the sentiment of the firstreview comment and the sentiment of the second review comment aredifferent; and generating, for each of the plurality of reviewers, aquestion set comprising a plurality of questions based upon a conflictidentified for a review comment of the corresponding reviewer, whereinthe corresponding reviewer answering the generated question resolves theconflict.
 2. The method of claim 1, wherein the categorizing comprisescategorizing each of the review comments into a high level review topicand further categorizing the review comments into more specific reviewtopics.
 3. The method of claim 1, wherein the categorizing comprisesusing a deep learning technique to classify phrases of the plurality ofreview comments into a review topic.
 4. The method of claim 1, whereinthe generating a question set comprises (i) detecting subjective reviewcomments within the categorized review comments and (ii) generating aquestion requesting more information related to the subjective reviewcomments.
 5. The method of claim 1, wherein the generating a questionset comprises (i) detecting, from the categorized review comments, areview comment from a given reviewer that needs to be validated byanother of the plurality of reviewers and (ii) generating a question,for at least one of the reviewers other than the given reviewer,requesting validation of the review comment.
 6. The method of claim 1,wherein the generating a question set comprises using a natural languageprocessing technique to generate a natural language question.
 7. Themethod of claim 1, comprising selecting a subset of the generatedquestion set, wherein the selected subset comprises the minimum numberof questions necessary for resolving the conflicts.
 8. The method ofclaim 1, comprising (i) validating the generated question set againstidentified parameters for the document and (ii) discarding questionsfrom the question set identified as irrelevant based upon the identifiedparameters.
 9. The method of claim 8, wherein the identified parametersare identified based upon a submission setting for the document.
 10. Themethod of claim 1, comprising (i) examining the review comments for ananswer to a generated question and (ii) discarding a question for whichan answer has previously been provided.
 11. An apparatus, comprising: atleast one processor; and a computer readable storage medium havingcomputer readable program code embodied therewith and executable by theat least one processor, the computer readable program code comprising:computer readable program code configured to receive a plurality ofreview comments from each of a plurality of reviewers tasked withreviewing a document; computer readable program code configured tocategorize, for each of the plurality of reviewers, each of theplurality of review comments into one of a plurality of review topics;computer readable program code configured to identify a conflict between(a) a first review comment categorized into a review topic and providedby one of the plurality of reviewers and (b) a second review commentcategorized into the same review topic and provided by another of theplurality of reviewers, wherein the identifying a conflict comprises (i)identifying a sentiment of the first review comment and a sentiment ofthe second review comment and (ii) determining that the sentiment of thefirst review comment and the sentiment of the second review comment aredifferent; and computer readable program code configured to generate,for each of the plurality of reviewers, a question set comprising aplurality of questions based upon a conflict identified for a reviewcomment of the corresponding reviewer, wherein the correspondingreviewer answering the generated question resolves the conflict.
 12. Acomputer program product, comprising: a computer readable storage mediumhaving computer readable program code embodied therewith, the computerreadable program code executable by a processor and comprising: computerreadable program code configured to receive a plurality of reviewcomments from each of a plurality of reviewers tasked with reviewing adocument; computer readable program code configured to categorize, foreach of the plurality of reviewers, each of the plurality of reviewcomments into one of a plurality of review topics; computer readableprogram code configured to identify a conflict between (a) a firstreview comment categorized into a review topic and provided by one ofthe plurality of reviewers and (b) a second review comment categorizedinto the same review topic and provided by another of the plurality ofreviewers, wherein the identifying a conflict comprises (i) identifyinga sentiment of the first review comment and a sentiment of the secondreview comment and (ii) determining that the sentiment of the firstreview comment and the sentiment of the second review comment aredifferent; and computer readable program code configured to generate,for each of the plurality of reviewers, a question set comprising aplurality of questions based upon a conflict identified for a reviewcomment of the corresponding reviewer, wherein the correspondingreviewer answering the generated question resolves the conflict.
 13. Thecomputer program product of claim 12, wherein the categorizing comprisescategorizing each of the review comments into a high level review topicand further categorizing the review comments into more specific reviewtopics.
 14. The computer program product of claim 12, wherein thegenerating a question set comprises (i) detecting subjective reviewcomments within the categorized review comments and (ii) generating aquestion requesting more information related to the subjective reviewcomments.
 15. The computer program product of claim 12, wherein thegenerating a question set comprises (i) detecting, from the categorizedreview comments, a review comment from a given reviewer that needs to bevalidated by another of the plurality of reviewers and (ii) generating aquestion, for at least one of the reviewers other than the givenreviewer, requesting validation of the review comment.
 16. The computerprogram product of claim 12, wherein the generating a question setcomprises using a natural language processing technique to generate anatural language question.
 17. The computer program product of claim 12,comprising selecting a subset of the generated question set, wherein theselected subset comprises the minimum number of questions necessary forresolving the conflicts.
 18. The computer program product of claim 12,comprising (i) validating the generated question set against identifiedparameters for the document and (ii) discarding questions from thequestion set identified as irrelevant based upon the identifiedparameters.
 19. The computer program product of claim 12, comprising (i)examining the review comments for an answer to a generated question and(ii) discarding a question for which an answer has previously beenprovided.
 20. A method, comprising: obtaining a plurality of documentevaluation comments from each of a plurality of evaluators correspondingto evaluation of a document; classifying each of the evaluation commentsinto one of a plurality of evaluation topics, wherein the classifyingcomprises extracting phrases from the evaluation comments andclassifying the extracted phrases; detecting, for each of the evaluationtopics, a disagreement between a first evaluation comment provided byone of the plurality of reviewers and a second evaluation commentprovided by another of the plurality of reviewers, wherein the detectingcomprises detecting a contrasting sentiment between the first and secondevaluation comments; generating, for each of the plurality ofevaluators, a minimal question set to resolve the disagreement betweenthe evaluation comments; and providing the minimal question set to thecorresponding evaluator.